feat(datafusion): add max_temp_directory_size parameter for z-order and compact operations for DataFusion #3847

fvaleye · 2025-10-14T09:05:50Z

Description

Add max_temp_directory_size parameter to the Python bindings to optimize DataFusion operations.
Add max_spill_size to the Python binding (available in Rust)

Related Issue(s)

closes Asked to increase max_temp_directory_size in the disk manager configuration when optimizing large table #3833

codecov · 2025-10-14T09:07:47Z

Codecov Report

❌ Patch coverage is 29.85075% with 47 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.63%. Comparing base (cb672ac) to head (6909bff).

Files with missing lines	Patch %	Lines
crates/core/src/delta_datafusion/session.rs	35.29%	22 Missing ⚠️
python/src/lib.rs	0.00%	14 Missing ⚠️
crates/core/src/operations/optimize.rs	47.05%	9 Missing ⚠️
python/src/query.rs	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3847      +/-   ##
==========================================
- Coverage   73.72%   73.63%   -0.09%     
==========================================
  Files         151      151              
  Lines       39250    39272      +22     
  Branches    39250    39272      +22     
==========================================
- Hits        28936    28919      -17     
- Misses       9005     9045      +40     
+ Partials     1309     1308       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ion-elgreco · 2025-10-14T09:11:30Z

crates/core/src/operations/optimize.rs

            preserve_insertion_order: false,
            max_concurrent_tasks: num_cpus::get(),
            max_spill_size: 20 * 1024 * 1024 * 1024, // 20 GB.
+            max_temp_directory_size: 100 * 1024 * 1024 * 1024, // 100 GB.


Maybe not expose it here if we immediately deprecate it, but rather create a proper configured Session in the python binding when calling optimize and passing that around

I took the simple approach.
I can take more time to create a properly configured session, something like this:

fn create_optimize_session( max_spill_size: usize, max_temp_directory_size: u64, ) -> SessionState {

WDYT?

Looks good!

Okey dokey!
I will refactor this to remove depreciation and configure a proper session. I will ping you when it's ready.

abhiaagarwal · 2025-10-14T11:40:28Z

I looked into this similarly and unfortunately, there doesn't seem to be a way to allow passing in a datafusion SessionState from the python side to the rust side, which would give python callers the maximum flexibility :/

ion-elgreco · 2025-10-14T12:27:04Z

I looked into this similarly and unfortunately, there doesn't seem to be a way to allow passing in a datafusion SessionState from the python side to the rust side, which would give python callers the maximum flexibility :/

Not a state, but you can consume FFISessionConfig, not sure if it has all the some buttons and knobs as state/context

abhiaagarwal · 2025-10-14T12:49:47Z

I looked into this similarly and unfortunately, there doesn't seem to be a way to allow passing in a datafusion SessionState from the python side to the rust side, which would give python callers the maximum flexibility :/

Not a state, but you can consume FFISessionConfig, not sure if it has all the some buttons and knobs as state/context

It accomplishes the goal on the per-operation level, but if you wanted to share a SessionContext across multiple operations (say, to limit the total amount of ram for working with multiple tables concurrently) you're out of luck.

I guess the rust side could store a SessionContext as an optional in RawDeltaTable and pass that in every operation to achieve the same goal, though. Definitely not done in this ticket, though :D

fvaleye · 2025-10-14T14:34:31Z

I looked into this similarly and unfortunately, there doesn't seem to be a way to allow passing in a datafusion SessionState from the python side to the rust side, which would give python callers the maximum flexibility :/

Not a state, but you can consume FFISessionConfig, not sure if it has all the some buttons and knobs as state/context

Something like:

config = SessionConfig()
config.set_config("datafusion.execution.target_partitions", "8")
config.set_config("datafusion.optimizer.skip_failed_rules", "false")

dt.optimize.z_order(
    columns=["id"],
    session_config=config,  # Advanced DataFusion config
)

We can have a follow-up PR for this, WDYT?

ion-elgreco · 2025-10-14T14:40:15Z

I looked into this similarly and unfortunately, there doesn't seem to be a way to allow passing in a datafusion SessionState from the python side to the rust side, which would give python callers the maximum flexibility :/

Not a state, but you can consume FFISessionConfig, not sure if it has all the some buttons and knobs as state/context

Something like:
config = SessionConfig()
config.set_config("datafusion.execution.target_partitions", "8")
config.set_config("datafusion.optimizer.skip_failed_rules", "false")

dt.optimize.z_order(
    columns=["id"],
    session_config=config,  # Advanced DataFusion config
)
We can have a follow-up PR for this, WDYT?

Yeah might be nice for interopability with datafusion-python

roeap · 2025-10-17T23:19:09Z

@fvaleye @ion-elgreco @abhiaagarwal - I just opened #3860 to create a more narrow waist when creating datafusion sessions. Going forward we also need to wire our object-store and log store factories with the datafusion session.

Would it be possible to base this PR on the one above to properly configure the session?

fvaleye · 2025-10-19T11:40:52Z

@fvaleye @ion-elgreco @abhiaagarwal - I just opened #3860 to create a more narrow waist when creating datafusion sessions. Going forward we also need to wire our object-store and log store factories with the datafusion session.

Would it be possible to base this PR on the one above to properly configure the session?

Yes, I will do that @roeap 👍

…nContext Signed-off-by: Florian Valeye <[email protected]>

github-actions · 2025-10-20T08:15:14Z

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

ion-elgreco · 2025-10-20T08:17:21Z

@fvaleye can you resolve the conflict, then we are good to go :)

fvaleye · 2025-10-20T08:20:46Z

@fvaleye can you resolve the conflict, then we are good to go :)

I'm on it!

…n-from-Python

fvaleye requested review from hntd187, ion-elgreco, roeap and rtyler as code owners October 14, 2025 09:05

github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels Oct 14, 2025

ion-elgreco reviewed Oct 14, 2025

View reviewed changes

fvaleye force-pushed the feature/configure-disk-manager-for-datafusion-from-Python branch from c82d4ba to 332179c Compare October 14, 2025 09:15

fvaleye force-pushed the feature/configure-disk-manager-for-datafusion-from-Python branch 4 times, most recently from b0d47cf to 79dcc10 Compare October 14, 2025 14:01

feat(optimize): integrate disk manager configuration with DeltaSessio…

5c83b44

…nContext Signed-off-by: Florian Valeye <[email protected]>

fvaleye force-pushed the feature/configure-disk-manager-for-datafusion-from-Python branch 2 times, most recently from 2fc4046 to 6652cef Compare October 20, 2025 08:14

ion-elgreco previously approved these changes Oct 20, 2025

View reviewed changes

fvaleye dismissed ion-elgreco’s stale review via 16e6111 October 20, 2025 08:18

fvaleye force-pushed the feature/configure-disk-manager-for-datafusion-from-Python branch from 16e6111 to 6652cef Compare October 20, 2025 08:19

fvaleye force-pushed the feature/configure-disk-manager-for-datafusion-from-Python branch from 4648d1b to 5c83b44 Compare October 20, 2025 08:25

Merge branch 'main' into feature/configure-disk-manager-for-datafusio…

6909bff

…n-from-Python

feat(datafusion): add max_temp_directory_size parameter for z-order and compact operations for DataFusion #3847

Are you sure you want to change the base?

feat(datafusion): add max_temp_directory_size parameter for z-order and compact operations for DataFusion #3847

Uh oh!

Conversation

fvaleye commented Oct 14, 2025

Description

Related Issue(s)

Uh oh!

codecov bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ion-elgreco Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

fvaleye Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

ion-elgreco Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

fvaleye Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

abhiaagarwal commented Oct 14, 2025

Uh oh!

ion-elgreco commented Oct 14, 2025

Uh oh!

abhiaagarwal commented Oct 14, 2025

Uh oh!

fvaleye commented Oct 14, 2025

Uh oh!

ion-elgreco commented Oct 14, 2025

Uh oh!

roeap commented Oct 17, 2025

Uh oh!

fvaleye commented Oct 19, 2025

Uh oh!

github-actions bot commented Oct 20, 2025

Uh oh!

ion-elgreco commented Oct 20, 2025

Uh oh!

fvaleye commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Oct 14, 2025 •

edited

Loading